add compact() API and auto-rebuild indexes #339

staltz · 2022-04-20T11:14:30Z

This PR adds the compact() ("async") API to ssb-db2 which just triggers async-append-only-log compact(). When it's done, it will rebuild all leveldb indexes from scratch, rebuild private indexes, and partially rebuild jitdb indexes. This is all done in a way that is crash-resistant, capable of resuming where it had stopped. And compaction is guaranteed to not happen concurrently with log.stream that builds indexes, or any jitdb query.

staltz · 2022-04-20T14:17:55Z

There's an error in the test compaction resumes automatically after a crash:

RangeError [ERR_OUT_OF_RANGE]: The value of "offset" is out of range. It must be >= 0 and <= 65534. Received 77082
    at new NodeError (internal/errors.js:322:7)
    at boundsError (internal/buffer.js:85:9)
    at Buffer.readUInt16LE (internal/buffer.js:242:5)
    at Object.readDataLength (/home/runner/work/ssb-db2/ssb-db2/node_modules/async-append-only-log/record.js:23:19)
    at Object.getDataNextOffset (/home/runner/work/ssb-db2/ssb-db2/node_modules/async-append-only-log/index.js:234:31)
    at Stream._handleBlock (/home/runner/work/ssb-db2/ssb-db2/node_modules/async-append-only-log/stream.js:89:37)
    at Stream._resumeCallback (/home/runner/work/ssb-db2/ssb-db2/node_modules/async-append-only-log/stream.js:145:27)
    at Object.getBlock (/home/runner/work/ssb-db2/ssb-db2/node_modules/async-append-only-log/index.js:201:7)
    at Stream._resume (/home/runner/work/ssb-db2/ssb-db2/node_modules/async-append-only-log/stream.js:136:12)
    at Stream._next (/home/runner/work/ssb-db2/ssb-db2/node_modules/looper/index.js:11:9) {
  code: 'ERR_OUT_OF_RANGE'
}

It happens mostly in CI, hard to reproduce locally. Have to solve this crash before the PR can be said to be ready.

staltz · 2022-05-01T06:56:04Z

I fixed this error, it was about updateIndexes() (and a log.stream()) happening concurrently with compaction.

When running compact() in ssb-db2, it checks whether there is any active updateIndexes() going on, postponing the compact if there are.

However, if we are resuming from a crash, then compaction is started automatically by async-append-only-log, and then updateIndexes() is also started automatically in ssb-db2. So I fixed this by adding an if condition in updateIndexes() which checks whether compaction is ongoing. This works because AAOL resumes compaction synchronously, while updateIndexes() is always started asynchronously (it waits for a promise to load, so we know that it happens later).

db.js

staltz · 2022-05-01T07:08:06Z

@arj03 Although this is the final task for ssb-db2, I have still 2 TODOs:

EBT: we need to update or delete some entries after we delete msgs from the log. To keep it simple, let's suppose we only use sbot.db.deleteFeed and thus we delete that whole feedId from EBT too
What about browser compatibility for AAOL/compaction.js and DB2 post-compact files?

db.js

arj03 · 2022-05-02T06:51:54Z

db.js

@@ -774,7 +879,6 @@ exports.init = function (sbot, config) {
    getState: () => state,
    getIndexes: () => indexes,
    getIndex: (index) => indexes[index],
-    clearIndexes,


This was something I used in browser core for testing. I don't mind removing this. It's a thin wrapper around indexes reset anyway.

Do you want to bring it back?

db.js

README.md

arj03

Nice work 🚀

staltz · 2022-05-02T13:22:36Z

Feels crazy to finally merge this, but here we go!

github-actions · 2022-05-02T13:30:11Z

Benchmark results

Part	Duration
add 1000 elements	499ms
add 1000 private box1 elements	1339ms
unbox 1000 private box1 elements first run	147ms
unbox 1000 private box1 elements second run	111ms
add 1000 private box1 elements	1364ms
query 1000 elements first run	40ms
query 1000 elements second run	27ms
add 1000 private box2 elements	823ms
unbox 1000 private box2 elements first run	429ms
unbox 1000 private box2 elements second run	441ms
Migrate (+db1)	14870ms
Migrate (alone)	4962ms
Migrate (+db1 +db2)	11202ms
Migrate (+db2)	7505ms
Migrate continuation (+db2)	1090ms
Memory usage without indexes	754.36 MB = 27.38 MB + etc
Initial indexing	705ms
Initial indexing maxCpu=86	4780ms
Initial indexing compat	1197ms
Two indexes updating concurrently	1060ms
key one initial	53ms
key two	5ms
key one again	0ms
reboot and key one again	51ms
latest root posts	660ms
latest posts	37ms
votes one initial	599ms
votes again	1ms
hasRoot	426ms
hasRoot again	0ms
author one posts	357ms
author two posts	19ms
dedicated author one posts	426ms
dedicated author one posts again	0ms
Maximum memory usage	811.09 MB = 45.74 MB + etc
Indexes folder size	9.97mb

staltz mentioned this pull request Apr 20, 2022

add APIs necessary for log compaction algorithms ssbc/jitdb#217

Merged

ssbc deleted a comment from github-actions bot Apr 20, 2022

staltz mentioned this pull request Apr 20, 2022

use sizeDiff=1 to represent holes filled ssbc/async-append-only-log#71

Merged

staltz added 4 commits April 29, 2022 16:27

add compact() API and auto-rebuild indexes

2dc9f9c

update jitdb

c299e53

there can only be one updateIndexes() at a time

314f780

post-compaction reindex also also resets state in memory

2d7be01

staltz force-pushed the compact branch from d9714ff to 2d7be01 Compare April 29, 2022 13:27

staltz added 2 commits April 30, 2022 23:42

tweak compaction tests: wait for delete flush

a05c38b

fix resume compact clashing with updateIndexes()

ceba3bc

staltz marked this pull request as ready for review May 1, 2022 06:50

staltz commented May 1, 2022

View reviewed changes

db.js Show resolved Hide resolved

document how plugin reset() works

7a43490

ssbc deleted a comment from github-actions bot May 1, 2022

staltz requested a review from arj03 May 1, 2022 07:08

arj03 reviewed May 2, 2022

View reviewed changes

db.js Show resolved Hide resolved

arj03 reviewed May 2, 2022

View reviewed changes

db.js Show resolved Hide resolved

remove promisify-4loc dep

fc6bc26

arj03 reviewed May 2, 2022

View reviewed changes

README.md Show resolved Hide resolved

add compact() to README

f252f89

ssbc deleted a comment from github-actions bot May 2, 2022

arj03 approved these changes May 2, 2022

View reviewed changes

ssbc deleted a comment from github-actions bot May 2, 2022

staltz merged commit 7689958 into master May 2, 2022

staltz deleted the compact branch May 2, 2022 13:24

staltz mentioned this pull request May 2, 2022

new forget() API ssbc/ssb-ebt#66

Merged

ssbc deleted a comment from github-actions bot May 5, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add compact() API and auto-rebuild indexes #339

add compact() API and auto-rebuild indexes #339

staltz commented Apr 20, 2022 •

edited

Loading

staltz commented Apr 20, 2022

staltz commented May 1, 2022 •

edited

Loading

staltz commented May 1, 2022

arj03 May 2, 2022

staltz May 2, 2022

arj03 left a comment

staltz commented May 2, 2022

github-actions bot commented May 2, 2022

add compact() API and auto-rebuild indexes #339

add compact() API and auto-rebuild indexes #339

Conversation

staltz commented Apr 20, 2022 • edited Loading

staltz commented Apr 20, 2022

staltz commented May 1, 2022 • edited Loading

staltz commented May 1, 2022

arj03 May 2, 2022

Choose a reason for hiding this comment

staltz May 2, 2022

Choose a reason for hiding this comment

arj03 left a comment

Choose a reason for hiding this comment

staltz commented May 2, 2022

github-actions bot commented May 2, 2022

Benchmark results

staltz commented Apr 20, 2022 •

edited

Loading

staltz commented May 1, 2022 •

edited

Loading